Pruning e2e verification by alrxy · Pull Request #404 · symbioticfi/relay

alrxy · 2026-03-25T07:27:45Z

PR Type

Tests, Enhancement

Description

Add pruning end-to-end verification test
Scan bbolt and badger storage
Parameterize retention, sync, pruner settings
Add CI job and Make target

Diagram Walkthrough

flowchart LR
  A["Pruning E2E test"] -- "creates epoch-bound request" --> B["Wait for signatures and proof"]
  B -- "waits until prune eligibility" --> C["Scan persisted sidecar data"]
  C -- "supports both backends" --> D["bbolt and badger storage"]
  E["Scripts and CI configuration"] -- "inject retention and pruner env" --> A

File Walkthrough

Relevant files

Tests

3 files

pruning_test.go `Add pruning end-to-end storage verification`	+419/-0
tests.yaml `Add pruning E2E CI matrix job`	+89/-0
Makefile `Add dedicated pruning E2E target`	+4/-0

Enhancement

1 files

types_test.go `Extract reusable environment loading helper`	+11/-3

Configuration changes

3 files

generate_network.sh `Pass pruning settings into generated network`	+16/-1
sidecar-start.sh `Make sidecar pruning values environment-driven`	+10/-5
setup.sh `Export pruning-related environment variables`	+6/-1

github-actions · 2026-03-25T07:29:07Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Incomplete coverage The new pruning check only scans `sidecarIndex` 0, even though the test creates the request across all sidecars and waits for signatures from all of them. If pruning is broken on any non-zero sidecar, this test will still pass and miss the regression. before, err = scanSidecarStorage(0, targetEpoch, common.HexToHash(requestID), nil) require.NoError(t, err) } t.Log("Step 5: Waiting for pruning and verifying that no non-excluded entities remain...") require.NoError(t, waitForErrorIsNil(ctx, pruningTimeout(), func() error { remaining, err := scanSidecarStorage(0, targetEpoch, common.HexToHash(requestID), excluded)

github-actions · 2026-03-25T07:30:53Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Verify every sidecar storage The test only inspects `sidecarIndex` `0`, so it can pass even when pruning is broken on the other sidecars. Iterate over every configured sidecar during verification; otherwise this e2e check misses cluster-wide pruning regressions. e2e/tests/pruning_test.go [87-103] +requestHash := common.HexToHash(requestID) + require.NoError(t, waitForErrorIsNil(ctx, pruningTimeout(), func() error { - remaining, err := scanSidecarStorage(0, targetEpoch, common.HexToHash(requestID), excluded) - if err != nil { - return err + var allRemaining []remainingEntity + + for i := range envInfo.GetSidecarConfigs() { + remaining, err := scanSidecarStorage(i, targetEpoch, requestHash, excluded) + if err != nil { + return err + } + for _, entity := range remaining { + entity.Location = fmt.Sprintf("sidecar-%02d/%s", i+1, entity.Location) + allRemaining = append(allRemaining, entity) + } + if debugScan && i == 0 { + after, scanErr := scanSidecarStorage(i, targetEpoch, requestHash, nil) + if scanErr != nil { + return scanErr + } + finalAfter = after + } } - if debugScan { - after, scanErr := scanSidecarStorage(0, targetEpoch, common.HexToHash(requestID), nil) - if scanErr != nil { - return scanErr - } - finalAfter = after - } - if len(remaining) > 0 { - return fmt.Errorf("found unpruned entities:\n%s", formatRemainingEntities(remaining)) + + if len(allRemaining) > 0 { + return fmt.Errorf("found unpruned entities:\n%s", formatRemainingEntities(allRemaining)) } return nil })) Suggestion importance[1-10]: 8 __ Why: This is a correct and important gap: `TestPruningE2E_RemovesAllNonExcludedEntities` currently verifies pruning only on sidecar `0`, so failures on other sidecars would go undetected. The suggested loop is consistent with how the request is created across all sidecars and materially improves the test's coverage.	Medium
Possible issue	Match both hex ID formats `requestID` matching only checks raw bytes and the `"0x"`-prefixed hex form, so any entries stored as plain hex will be missed and the test can report a false pass. Match both prefixed and non-prefixed hex encodings to avoid silently skipping unpruned records. e2e/tests/pruning_test.go [336-347] func matchesPrunedEntity(key, value, epochBytes, requestIDBytes []byte, requestIDHex string) bool { + requestIDHexNoPrefix := strings.TrimPrefix(requestIDHex, "0x") + if bytes.Contains(key, requestIDBytes) \|\| bytes.Contains(value, requestIDBytes) { return true } if bytes.Contains(key, []byte(requestIDHex)) \|\| bytes.Contains(value, []byte(requestIDHex)) { + return true + } + if bytes.Contains(key, []byte(requestIDHexNoPrefix)) \|\| bytes.Contains(value, []byte(requestIDHexNoPrefix)) { return true } if bytes.Contains(key, epochBytes) \|\| bytes.Contains(value, epochBytes) { return true } return false } Suggestion importance[1-10]: 5 __ Why: This is a plausible correctness improvement because `matchesPrunedEntity` currently checks raw bytes and only the `0x`-prefixed hex form from `requestID.Hex()`. If some stored keys or values use hex without the prefix, the scan could miss leftover records and falsely pass.	Low
General	Wait for healthy services A fixed `sleep 30` makes the pruning job flaky because slower runners can still be starting containers when the tests begin. Wait for the compose services to become healthy instead of assuming startup always fits in 30 seconds. .github/workflows/tests.yaml [422-427] - name: "Start Pruning Test Network" run: \| cd e2e/temp-network docker compose up -d - sleep 30 + timeout 180 bash -c ' + until ! docker compose ps \| grep -Eq "starting\|unhealthy\|created"; do + sleep 5 + done + ' + Suggestion importance[1-10]: 6 __ Why: Replacing a fixed `sleep 30` with an actual readiness wait is a relevant improvement for CI stability, especially since container startup time varies across runners. The proposed change aligns with the defined healthchecks and reduces flaky failures before `make e2e-pruning-test` runs.	Low

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6df10e3415

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-03-25T07:35:43Z

+			Message:       []byte(msg),
+			RequiredEpoch: (*uint64)(&epoch),
+		})
+		require.NoErrorf(t, err, "failed to sign message on sidecar %d", i)


Retry SignMessage when creating pruning test requests

createPruningRequest performs a single SignMessage call per sidecar and immediately fails on any transient RPC/storage error. This makes the new pruning test flaky under the same transaction-conflict conditions already handled elsewhere in this suite (for example, sign_test.go retries this API). A bounded retry loop here would prevent intermittent CI failures that are unrelated to pruning behavior.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-03-25T07:35:43Z

+RETENTION_SIGNATURE_EPOCHS=${RETENTION_SIGNATURE_EPOCHS:-1000}
+RETENTION_PROOF_EPOCHS=${RETENTION_PROOF_EPOCHS:-1000}
+PRUNER_INTERVAL=${PRUNER_INTERVAL:-1m}
+SYNC_EPOCHS=${SYNC_EPOCHS:-1000}


Derive sync epochs from retention defaults

SYNC_EPOCHS defaults to 1000 even when retention is overridden to a much smaller value. With this change, users can now set RETENTION_VALSET_EPOCHS via env, but if they forget to also set SYNC_EPOCHS, the generated config becomes invalid (sync.epochs > retention.valset-epochs) and relay startup fails due to config validation. The default should be capped/derived from retention to avoid this regression when enabling pruning settings.

Useful? React with 👍 / 👎.

github-actions · 2026-03-25T07:38:41Z

🧪 Test Coverage Report

Coverage: 50.7%

# Conflicts: # .github/workflows/tests.yaml # e2e/scripts/generate_network.sh

test: prune e2e

6df10e3

github-actions Bot added the Review effort 3/5 label Mar 25, 2026

fix: lint

75ff81f

chatgpt-codex-connector Bot reviewed Mar 25, 2026

View reviewed changes

alrxy and others added 12 commits March 26, 2026 14:23

fix: badger scan

7e8c63e

fix: pruning test on non-fresh setup

a45444b

Merge branch 'main' into prune-verification

14474d5

fix: access to badger file

2f2ffa1

fix: chmod before stop

5f2fe11

fix: offline scan for both dbs

4f26b27

fix: remove waiting for no error before scanning

8fe8824

fix: adjust pruning envs

01962c2

fix: test stop and chmod

893c936

Merge branch 'main' into prune-verification

7126f58

fix: change root user

f612e19

chore: add prescan in pruning and extend time limit to 40m

32e393d

alrxy requested a review from oxsteins April 2, 2026 07:00

Merge remote-tracking branch 'origin/main' into crazy-goldstine-a9b85a

2f3eee4

# Conflicts: # .github/workflows/tests.yaml # e2e/scripts/generate_network.sh

alrxy requested a review from ilyasymbiotic May 15, 2026 07:15

Merge branch 'main' into prune-verification

785c3a4

Navigation Menu

Uh oh!

Pruning e2e verification - #404

Pruning e2e verification#404
alrxy wants to merge 16 commits into
mainfrom
prune-verification

alrxy commented Mar 25, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Mar 25, 2026

Uh oh!

chatgpt-codex-connector Bot Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

alrxy commented Mar 25, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Type

Description

Diagram Walkthrough

File Walkthrough

Uh oh!

github-actions Bot commented Mar 25, 2026

PR Reviewer Guide 🔍

Uh oh!

github-actions Bot commented Mar 25, 2026

PR Code Suggestions ✨

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 Test Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

alrxy commented Mar 25, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Mar 25, 2026 •

edited

Loading